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1 Introduction 

Let F — F[xi, X2t ■ ■) be an absolutely free group with basis xi,X2t ■ ■ ■ ■ Recall that a group H satisfies 
an identity w{xi , . . . , Xn) = 1 for a word w{xi , . . . , x„) = w G if w vanishes under every honiomorphism 
F ^ H. A variety of groups is a class of groups consisting of all groups that satisfy some set of identities. 

A relatively free group G is a free group in a group variety V, i.e., G belongs to V, and G is generated 
by a set Y such that every mapping Y H, where iJ € V, extends to a homomorphism G —> H. The 
(free) rank of G is the cardinality of Y. (See details in [N].) We denote by F„, by = Fn/F^^, and 
by M„ = Fn/F^ the absolutely free group, the free abelian and the free metabelian groups of rank n, 
respectively. 

The group of inner automorphisms of a group G is normal in Aut{G), and so the factor group G/Z{G), 
where Z{G) is the center of G, can be canonically identified with a normal subgroup of Aut{G). Since 
the center of M„ is trivial for n > 2 (see [N], 25.63), one can identify M„ with the normal subgroup of 
Aut{Mn) consisting of the inner automorphisms of M„. 

Our paper is inspired by the following result of V.P.Platonov that answers a question raised by 
H.Mochizuki [M^. 

Theorem 1.1. (V.P.Platonov [PI) (1) Let p he a finite- dimensional linear representation oj the auto- 
morphism group Aut{Mn) over a field k. Then the image p(M„) is a virtually nilpotent group. (2) It 
follows that the group Aut{Mn) is not linear for n> 1. 

As usual, a group G is called linear if it is isomorphic to a subgroup of GLm{k) for some field k and 
some integer m > 1. 

In Section 2, we give an alternative and shorter proof of Theorem ll.il Then a similar approach and 
the utilization of some known properties of group varieties lead to the complete description of relatively 
free groups G for which Aut(G) is a linear group. 

Theorem 1.2. Let G he a relatively free hut not ahsolutely free group. The automorphism group Aut{G) 
is linear if and only if G is a finitely generated virtually nilpotent group. 

Furthermore, if the group G is finitely generated but not virtually-nilpotent, then there is an automor- 
phism (p of G such that the extension P of G/Z{G) hy (f> is a non-linear suhgroup of Aut{G); and if G is 
finitely generated and virtually nilpotent, then the holomorph Hol{G) is linear over Z. 

Recall that a group G is virtually nilpotent if it contains a (normal) nilpotent subgroup of finite index. 
(The "if part of the statement does not need the hypothesis that G is relatively free.) 



Remark 1.3. The automorphism group Aut{Fn) is not linear for n > 3 (Formanek, Procesi |FPj ) but 
the group Aut{F2) is linear (Krammer pr ). 



Remark 1.4. Note that the formulation of Theorem 11.21 is similar to those contained in the papers of 
O.M.Mateiko and O.I.Tavgen' [MT^ and A.A.Korobov [Koj . Nevertheless we prove Theorem 1 1 . 2 1 here for 
the following reasons. (1) There is a mistake in both [MTj and |Ko| . Namely, the proofs essentially use 
the "known property" of Fitting subgroups to be fully characteristic. But this does not hold even for 
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relatively free groups. (For example, the Fitting subgroup is not fully characteristic in the free group of 
rank n > 1 of the variety generated by the alternating group Alt{5).) (2) The formulation of the main 
theorems is not quite correct in both jMTj and [Ko| because it is not proved there that the virtually 
nilpotency of a free group of rank n > 1 in a variety implies virtually nilpotency of free groups having 
rank > n. (3) Our proof is simpler. (The authors of both papers [MTj and [Ko| refer, in particular, to the 
statement that all locally finite groups of exponent dividing m form a variety. This claim is equivalent 
to the restricted Burnside problem for groups of exponent to, and the affirmative solution is bases on the 
Classification Hypothesis for finite simple groups.) 

2 Free matabelian case 

The following Kolchin-Mal'cev Theorem is the most-known fact on linear solvable groups. 

Lemma 2.1. Every linear solvable group has a subgroup H of finite index such that the derived subgroup 
H' is nilpotent. □ 

Let (j) be an automorphism of the free abelian group v4„ and = (A^, (j)) the extension of by the 
automorphism (f). Assume that no root of the characteristic polynomial of 4> is an k-th root of 1 for any 
integer fc > 0. The following property of the group is folklore. 

Lemma 2.2. Let C be a subgroup of finite index in B^. Then the derived subgroup C has finite index 
in A„. 

Proof. Since C is of finite index in S^, it must contain and mAn for some positive integers to and 
k. (We use the additive notation for A„ in this proof.) Therefore C contains the subgroup [mAn,4>''] = 
{0'^(TOa) — ma \ a G A,^}. 

Proving by contradiction, assume that the index (An '■ [mAn, 4''^]) is infinite. Then the image of toA„ 
under the mapping 0'^ — id is of rank < n. Hence is an eigenvalue of — id, and so A'^ = 1 for a 
characteristic root A of a contradiction. □ 

Proof of Theorem \l.l[ There is nothing to prove if n = 1. For every n >2, there exists an automor- 
phism (p of An whose characteristic roots are not roots of 1. (One can easily find such automorphisms 
for n — 2,3 and note for n > 3 that An is a direct sum of subgroups isomorphic to A2 and ^3.) We 
keep the same notation (p for a lifting of (j) to Aut{Mn). (Recall that An — M„/M^, and the induced 
homomorphism Aut{Mn) Aut{An) is surjective by [N], 41.21.) 

Let P = {Mn, 4>) be the extension of M„ by the automorphism (j). This P is a subgroup of Aut{Mn) 
since (0) n Af„ = {1}; and P is solvable because the factor-group P/Mn is cyclic. By Lemma [2Tl there 
is a normal subgroup T of finite index in P such that the subgroup p{T') is nilpotent. 

The canonical image C = TM'n/M'n of T in B^ = PjM'n has finite index, and therefore C" is of finite 
index in An — M„/M4 by Lemma Yl72\ Hence the inverse image D = T'M'n is of finite index in Mn- 

The normal subgroup p{Mn) of p{Mn) is abelian since M„ is metabelian. Thus, p{D) is nilpotent 
being a product of two nilpotent normal in p{P) subgroups p{T') and p{Mn). Since (M„ : _D) < 00, the 
theorem is proved. □ 

3 Few lemmas on varieties of groups 

The product lAV of two group varieties contains all the groups G having a normal subgroup N such that 
N (^U and G/N e V; UV is also a group variety ([N|, 21.12). 

Lemma 3.1. (fSh^ ). Let L be a free group of rank n > 2 in a product of varieties UV. Assume that the 
free group of rank n in the variety V is infinite and lA contains a non-trivial group. Then the center of L 
is trivial. □ 

We denote by A (by Ak) the variety of all abelian groups (of all abelian groups of exponent dividing 
k), and denote by Mk^n the free group of rank n in the variety Aik ^ AkA. 
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Lemma 3.2. If n,k > 2, then the group Mk,n is not virtually nilpotent. 

Proof. The wreath product W = Zi^wrZ, of a cyclic group of order k and an infinite cycUc group is 
2-generated, and it belongs to the variety A4k- Therefore is a homomorphic image of the group M^ n- 
Since W is not virtually nilpotent, Mj, „ is not virtually nilpotent too. □ 

A variety V is called solvable if all the groups of V are solvable. 

Lemma 3.3. (f^)- Let S be a solvable variety of groups. Then either S contains as a subvariety the 
product A4p = ApA for some prime p or every finitely generated group in S is virtually nilpotent. □ 

Further we call a variety V proper if it does not contain all groups, or equivently, the absolutely free 
group F2 does not belong to V. It is easy to see that a product of two proper varieties is proper. (See also 
[N].) The minimal variety containing a group Q is denoted by varQ. Given a group G and a variety V, 
the verbal subgroup V{G) corresponding to V is the smallest normal subgroup N of G such that G/N e V. 

4 Proof of Theorem 11.21 

By Auslander - Baumslag's theorem [AB| . the holomorph of any finitely generated (virtually) nilpotent 
group G is linear over Z. Thus, it remains to consider a non-virtually-nilpotent free group G of rank 
n > 2 in a proper variety V and construct the required automorphism (jj. (A non-trivial relatively free 
group of infinite rank admits the automorphisms from an infinite symmetric group that is not linear.) 
Now the quotient H = G/Z{G) is non-virtually-nilpotent normal subgroup of Aut{G). We may assume 
that H is a linear group since otherwise there is nothing to prove. 

Since both G and H satisfy a non-trivial identity and H is linear, the group H is virtually solvable by 
Platonov's theorem [Plj . i.e., the solvable radical R oi H is of finite index in H. Therefore i? is a finitely 
generated but not virtually nilpotent solvable group. 

By Lemma l3.3( there is a prime p such that A4p C varR C varH C varG. 

Therefore there are canonical epimorphisms G — > -Mp „ and G — > A„. The kernels are the verbal 
subgroups of G corresponding to the varieties Mp and A, respectively. The latest kernel is just the 
derived subgroup G", and we denote by Mp{G) the former one. 

The center of Mp_„ is trivial by Lemma [3Tl and so Z{G) is contained in Mp{G) C G' . Consequently, 
we have isomorphisms G/Mp{G) ~ H/Mp{H) ~ Mp,„ and G/G' ~ H/H' ~ A^. 

Now, as in the proof of Theorem 11.11 we introduce an automorphism cj) of An whose action has no 
characteristic roots equal to any root of 1. As there (by [N], 41.21) one can lift (p to Aut{G) and also to 
Aut{H) since the center Z{G) is a characteristic subgroup of G. Denote by P = {H, (j>) the extension of 
H by the automorphism (j). It is a subgroup of Aut{G) as in the proof of Theorem ll.il 

Proving by contradiction, assume that P is a linear group. Since P G VA, the group P satisfies a 
non-trivial identity, and by [Plj . P must have a solvable normal subgroup of finite index. By Lemma l2.H 
P contains a normal subgroup T of finite index with nilpotent derived subgroup T'. Applying Lemma 
Oto the image of T in P^ = P/H', we have {H : {T'H')) < oo. 

The quotient H' / Mp{H) is an abelian normal subgroup of H/Mp{H) ~ Mp^n- Since T' is nilpotent 
and normal in P, the image of the subgroup T'H' under the canonical epimorphism H — > Mp „ is nilpotent 
too. But this image is of finite index in Mp_„ because {H : {T'H')) < oo. This contradicts the statement 
of Lemma [321 and so the theorem is proved. □ 

Acknov^rledgments. The author is grateful to V.D.Mazurov, V.A.Romankov, N.S.Romanovskiy, and 
A.E.Zalesski for useful discussion. 
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